1. Field of the Invention
The present invention relates to system verification test (SVT) and, more specifically, to a virtual testbed containing emulated test results for use with test sequences of SVT on a System Under Test (SUT).
2. Description of the Related Art
Most systems, whether it is a hardware system or a software system, requires quality assurance (QA) and system verification tests (SVT) before it is released for actual use by the public. It is preferable to automate SVT, so that the SVT process can be carried out efficiently and accurately.
Software test automation in many cases requires that a testing program operate like a human interacting with a command-line interface (CLI) via protocols such as telnet, SSH (Secure Shell), or via a serial port. FIG. 1 illustrates a conventional SVT system interacting with the SUT (testbed). The testing system 100 (such as a test program) sends requests 140 containing commands to a testbed 120 such as the SUT to perform a configuration step in the test or to extract information from the SUT 120. In response to the requests 140, the testbed 120 sends responses 160 to testing system 100. The responses 160 are typically text, formatted in a way intended for human operators to digest.
Test of systems involves communication with many devices using a variety of different protocols (such as telnet, SSH, SNMP, web, etc.). When developing automated tests, it is necessary to have access to the actual equipment (SUT 120) so that the tests can be exercised on the actual equipment and therefore developed incrementally as the automated tests are developed. However, such test equipment is often very expensive or is of limited availability both in number and time, because such test equipment is often expensive prototypes. Also, a test developer may want to work offline outside of a firewall without connection the SUT 120. Moreover, it is often necessary to begin test automation and development of the test program before the hardware or software SUT 120 is even available for use in test development. Finally, another challenge is that automated tests are, themselves, difficult to verify, especially in the case of negative testing, when it is necessary to ensure that the automated test will correctly detect certain faults that are difficult to cause in the real equipment. It is difficult and costly to provide these fault conditions with an actual SUT 120.
Most automated tests today are scripts. These scripts use various libraries for communicating with the devices of the SUT 120. When developing the automated tests, one can try to remove code that would be communicating with real devices when there is no access to those actual devices. Also, one could add code in the test script itself to inject artificial faults. But such manipulation of code of the test script is difficult and time-consuming and in many cases will not properly exercise the test code.